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Several new programs have been added to those of the 
Comprehensive Occupational Data Analysis Programs (CODAP) , all 
oriented toward analyzing and manipulating information describing 
work tasks, rather than jobs or persons, REXAL1 analyzes the 
inter-rater agreement among judges concerning task-factor ratings. 
TSKFAC adds factor weight vectors to the JOB DESCRIPTION file. FACSTD 
generates a file of rater background information and raw ratings. 
FACSPC computes, displays, and saves mean factor ratings for 
identified groups of raters, COHGEM generates and saves factors and 
factor composites through user-specified operations. TSKCOE extracts 
vectors from the JOB DESCRIPTION file and computes intercorrelations 
and user-specified regression equations. PREFAC applies eguation 
weights and produces a predictor score vector. TSKCAT generates a 
user-specified vector of 1«s and O^ to identify tasks of interests 
FACSOH selects, sorts, and prints task-factor information, and 
computes new vectors based on differences between vectors, maximums 
or minimums of vector sets, cumulative percentages, and categories of 
tasks developed by six arithmetic operators. These programs fill an 
important gap by equipping the analyst with tools for addressing new 
problem areas. (Author/BH) 
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I am making the assumption that most of your here are fam- 
iliar with the CODAP analysis system, which is a comprehensive 
set of computer programs for analyzing and reporting occupa- 
tional information collected with job inventories. The CODAP 
system presently contains approximately thirty-five main pro- 
grams which are oriented toward producing task-level descrip- 
tions of the work performed by individuals and groups of 
individuals. The system also contains a number of programs for 
identifying and describing the types of jobs existing in an 
occupational category, and for describing the characteristics ' 
of individuals falling into special or job-type groups. Today, 
I would like to describe a new series of programs recently 
added to the CODAP system which. are oriented toward analyzing 
and manipulating information describing work. tasks, rather than 
jobs or persons. These programs fill an important gap by 
equipping the analyst with tools for addressing new problem 
areas. * 



, , ^scribing, the new programs, I should recognize 

that CODAP already contains some programs which could be 
considered as being task-oriented. For example, ASFACT is 
designed to summarize task-xactor rating information provided 
by job incumbents,, while GRPSCM can be applied to display the 
percent of members in various groups performing any given 
task. AVALUE and AVGPCT can be used to compute such indexes 
as the average grade level or time-in-service of individuals 
performing each work task in an inventory. Two other task- 
oriented programs in the UNIVAC 1108 version of CODAP are 
REXALL and TSKNDX. These will be described briefly, since 
they interact with the new programs now being released. 

REXALL is a very powerful and flexible program for ana- 
lyzing the inter-rater agreement among judges concerning task- 
factor ratings. This program also generates a task-factor 
weight deck for input into other CODAP programs. An example 
output from a REXALL run is given in Table 1. This table 
displays certain information concerning task-factor ratings 
provided- by each of 38 raters participating in a recent task 
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Table 1. REXALL R3ter Correlations with Sample 
Means — Raw Responses 
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difficulty analysis. The program first computes the man 
rating for each task across all raters; then it correlates 
the ratings provided by each particular racer with the vector 
of mean ratings obtained from the total sample. These cor- 
relations range from .61 to .81. The values provided by 
rater #7 correlate -.47 with tae ratings provided by the com- 
plete sample. He used the rating scale upside down. This is 
not unusual. We have analyzed task-factor information from 
hundreds of raters on a variety of task factors, and we find 
that about 5% of the participants reverse the rating scale. 
Rater #11 is "non-cooperative. 11 It is inconceivable that 
there would be no agreement between a supervisor and his peers 
concerning the relative difficulty of tasks in his occupation. 
This is an example of an individual who simply "filled in the 
squares 1 ' with random numbers. Both of these individuals con- 
tribute "error 11 variance to the mean factor vector, and, at 
AFHRL, we would elect to discard them from the data set. 
Another source of error is reflected in column 4, which reports 
the mean of ratings provided by each rater. In this particu- 
lar study, the raters were asked to rate the relative 
difficulties of tasks in their occupation using a 7-point 
scale. The values in column 4 reveal that some raters tended 
to use the upper part of the scale, while others tended tc use 
the lower part. For example, rater #6 had a mean of 2.96, 
while^rater #10 had a mean of 5.09. Since all ratings are 
relative, there is no reason why the moan rating across all 
tasks should vary significantly from r: rar to rater. Such 
differences in means introduce within-task variance, which is 
treated as error variance in computation of the inter-rater 
agreement coefficient. REXALL provides a method of elimina- 
ting such error variance by standardizing ratings. 

Table 2 presents summary data associated with the first 
REXALL run in the example. Notice that the reliability 
coefficient for the means based upon all 38 raters is .939, 
which is not bad, considering that: (a) two raters were 
included who reversed the rating scale; (b) one rater was non- 
cooperative; and (c) the ratings were not standardized. The 
sample value is only .288, which can be loosely defined as 
indicating the reliability for a single rater. It is this 
value which is entered into the Spearman-Brown prophecy 
formula to estimate the reliability of mean ratings based upon 
a sample of any given size. 

You will notice that REXALL provides a printout of all 
data entering into the computation of the reliability coef- 
ficient. It also indicates, among other things: the average 
number of ratings per task; number of raters; number of tasks; 
total number of task ratings; percent of possible ratings; 
mean and standard deviation of all ratings; mean and standard 



268 



4 



Table 2. Int^r-Rater Reliability Based on K;jw Hating 
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deviation of task means; and mean and standard deviation of 
task standard deviations. 

Normally, REXALL is executed at least twice to process 
data for a particular- factor in a study. First, it is run on 
raw data in order to identify non-cooperative and upside down 
raters for discard, in the second run, the values from the 
retained ra.^rs are standardized, unless an absolute scale is 
being used. Table 3 presents the summary information for the 
second run of the example data. Notice that three raters were 
discarded, leaving 35 raters in the run. The raw scores for 
each rater and for the means across all iters were standar- 
dized to a mean of 5.00 and a standard aviation of 1.00. 
These actions raised the reliability of the mean ratings from 
.939 to .973, even though the number of raters was reduced. 
At the same time, Lhe sample value was raised from .288 to 



One of the outputs from the REXALL is a weight deck con- 
taining the mean rated value for each task in the job inven- 
tory on the factor in question. This weight deck is in proper 
format for direct input into other CODAP programs. For 
example, mean task difficulty weights might be entered into 
the VARGEN program to compute the average difficulty of tasks 
performed by each worker in a sample. Normally, however, the 
weight deck is first entered into the TSKNDX program in order 
to display the mean ratings of each task, along with other 
relevant information. Table 4 presents an abbreviated output 
from a TSKNDX run for tasks in the Electrical Power Production 
career ladder. In this specialty, draining condensate from 
air receiver tanks is considered to be the simplest task, 
while repairing electronic governors is considered to be the 
most complex. The TSKNDX program is flexible. At the 
requestor's option, it will publish data in task order or 
from low to high or high to low on mean factor ratings. It 
also presents other data of interest, such as the number of 
workers in a sample of interest performing each task, as well 
as the percent of time being spent on each task. 

ASFACT, AVALUE, AVGPCT, REXALL, and TSKNDX are task- 
oriented programs which have been available for a number of 
years in the. IBM 7040 and UNIVAC 1108 versions of CODAP. All 
five of these programs are currently coded in FORTRAN V, and 
could easily be converted for execution on other computers. 
While these programs are all very useful and efficient, 
recent studies in job evaluation and curricula development 
have required analyses and report formats of task-factor data 
which could not be serviced by existing CODAP programs. For 
this reason, a new series of task-oriented programs has been 
written to fill the gap. These are enumerated in Table 5. 
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Table 3. Inter-Rater Reliability Eased on Adjusted Ratings 
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Table 5. New CODAP Programs for Analyzing and Displaying 
Task-Oriented Data 



TSKFAC - Adds factor weight vectors to JOB DESCRIPTION file. 

FACSTD - Generates file of rater background information and 
raw ratings. 

FACSPC - Computes, displays, and saves mean factor ratings 
for identified groups of raters. 

COMGEN - Generates and saves factors and factor composites 
through user-specified operations. 

TSKCOR - Extracts vectors from JOB DESCRIPTION file, computes 
inter-correlations and user-specified regression 
equations. 

PREFAC - Applies equation weights developed by TSKCOR, or 
CORREG, and produces predictor score vector. 

TSKCAT - Generates user-specified vector of I's and 0 f s to 
identify tasks of interest. Provides input to 
TSKFAC. 



FACSUM - Flexible program for selecting, sorting, and 

printing task-factor information. Computes new 
vectors based on differences between vectors, 
maximums or minimums of vector sets, cumulative 
percentages, and categories of tasks developed 
by six arithmetic operators. 
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The first program is called TSKFAC . It is used to create 
new records on the JOB DESCRIPTION file which represent 
values for task factors. Decks containing factor values may 
be keypunched. However,* such decks are normally generated by 
other COD/P programs, such as AVALUE, AVGPCT, REXALL, TSKGRP, 
or TSKNDX. The TSKFAC program produces Request cards, which 
can serve as control cards in AUTOJT, MTXPRT, FACSUM, and 
PRIJOB. Several factors can be written on the JOB DESCRIPTION 
file with a single execution of TSKFAC. 

The second program, labeled FACSTD, serves the same 
function as the INPSTD does, for job survey raw data in the 
present CODAP system, except it is used to create a Rater 
History file instead of a worker history file. Input to this 
program is the SETCHKed raw data file, which contains rating 
and history information. Input also includes Duty/Task title 
cards and History variable definitions. In the UNIVAC 1108 
version, FACSTD will accept 20,000 raters (a highly unlikely 
encounter), 1700 task ratings, 1000 history variables, and 26 
duty variables. 

FACSPC calculates and publishes r^w and standardized 
factor means for special groups of raters whose membership is 
defined in terms of some combination of background or computed 
variables. The mean ratings may be reported in high to low 
sequence on mean values, or on the percent of members performing, 
or in duty-task sequence. An option is provided for writing the 
output on the REPORT and/or JOB DESCRIPTION files. In some 
ways, FACSPC parallels the J0BSPC program, except that it 
applies to factor ratings instead of job descriptions. 

C0MGEN is a new program which primarily is used to 
generate composite factors in accordance with user specifica- 
tions. However, this program is very general-purpose, opera- 
ting on factor data in somewhat the same manner as PR0GEN 
currently operates on history and task data. The C0MGEN 
program generates coding from user-supplied verbs and FORTRAN 
statements to perform various mathematical operations on 
vectors from the JOB DESCRIPTION file. 

TSKC0R extracts up to 100 vectors from a CODAP JOB 
DESCRIPTION file and computes matrices of intercorrelations 
as well as user-specified regression problems. The correla- 
tion matrices may be saved on tape for future use by C0RREG. 

PREFAC accepts equation weights developed by TSKC0R or 
C0RREG and produces a predicted score deck which is ready for 
input by TSKFAC. 

10 
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TSKCAT is a simple program whi<:h translates card decks 
containing task numbers to card decks with the identified 
trsks represented as "l's" in a NTASK— long 'string (vector) 
of l's and O f Sr The output deck is designed for input into 
TSKFAC and other programs for later use as an operator vector. 
For example, it can be used to suppress information in a 
FACSUi-i renort. 

FACSUM is the most flexible program in the new task- 
oriented package. This program allows the user to print any 
of the vectors on the JOB DESCRIPTION file. However, during 
the same run, the user can generate and publish new vectors 
representing differences between two vectors; the maximums or 
minimums of se s of vectors; cumulative percentages; or cate- 
gories of tasks using any one of six arithmetic operators. 
Inclusion of sequence numbering, blank columns, sorting, and 
print suppression are under user control. 

Now that the new programs have been enumerated, let's go 
back through and add a few details. First, it should be 
recognized that all of the new programs are now an integral 
part of the CODAP system. They interact With each other and 
with CODAP programs in the previous version. FACSUM can 
extract, manipulate, and display any vectors on the JOB 
DESCRIPTION file, including the "percent performing," "time 
spent by those performing," and "time spent by total group" 
vectors associated with any previously computed job descrip- 
tion, lask-factor vectors generated by five previously 
available CODAP programs can be written on the JOB DESCRIPTION 
file by the TSKFAC program. Task factors placed on the JOB 
DESCRIPTION file can be manipulated by five previously avail- 
able CODAP programs. Output from the TSKCOR program can be 
addressed by the CORREG program. Reports generated by three 
of the new programs can be written on the CODAP REPORT 
file and later extracted and published using request cards. 

Now let's go through a semi-hypothetical example to 
illustrate how some of the task-oriented CODAP programs might 
be used in a study. To provide a degree of linkage with the 
paper being presented by Lt Col Mead, I will describe a study 
in which the goal is to establish training priorities for 
police patrolman tasks in the civiliar doctor. We will assume 
that supervisory ratings have been collected on three task 
factors which have relevance for our decision. These are (a) 
consequences of inadequate performance of each task, (b) task 
delay tolerance, and (c) task learning difficulty, 

A job inventory containing all of the tasks performed by 
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individuals in the police patrolman career ladder has been 
administered to a large sample of workers. Instructors in 
the training academy have indicated those tasks which are 
currently being taught. Our goal is to recommend for con- 
sideration certain modifications in course content and to 
, support these recommendations. Figure 1 presents a simplified 
schematic of one possible rout! to the solution. In Step 1, 
we will process the supervisory task ratings on the conse- 
quence, delay tolerance, and difficulty factors. This could 
be accomplished by going through the REXALL and TSKFAC pro- 
grams. The mean task ratings for all three factors could be 
written on the JOB DESCRIPTION file with a single TSKFAC run. 
An alternative route would be to go through the FACSTD and 
FACSPC programs, but this would not provide the basis for iden- 
tifying raters for elimination or computing reliability coef- 
ficients. However, the latter route would enable the investigator 
to compare ratings provided by subgroups. For example, he could 
compute the mean ratings pro/ided by police lieutenants and 
sergeants using FACSPC, and then display differences in their 
perspectives using the FACSUM program. A new program, called 
REXSPC, is being written to facilitate communication between 
the lACSTD and REXALL programs. Be that as it may, either of 
the two routes shown on the schematic can be used to get the 
mean factor ratings for the total sample onto the JOB DESCRIP- 
TION file. 

Since all of the information we have on the course cur- 
riculum is a vector of l"s and O's indicating whether each task 
is or is not in the present course, we simply punch this 
information into a card deck and enter it onto the JOB DESCRIPTION 
file using the TSKFAC program. This is represented in the sche- 
matic as Step 2. 

In Step 3, we generate two vectors from the job survey 
data file. First, we compute a job description using the 
JOBSPC program for patrolmen who have been on the job less 
than 12 months si*>ce graduation from the academy. The "percent 
performing 11 vector in this description car used to estimate 
the probability that academy graduates w J 1 be required to 
perform each task during their first ye ^ on the job, which is 
one of the most valuable factors for nv.icing curriculum 
decisions. 

The second factor generated from the survey data file is 
designed to identify tasks which are not relevant for con- 
sideration for inclusion in the academy course, since they are 
inappropriate for assignment to privates and corporals. In 
this case, apply the AVGPCT program to the grade variable. 
This program computes the sum of the cross-products of percent 

12 

276 



product of many years of curriculum judgments. Our goal 
o capture the systematic variance in this judgment process 

regression equation which gives consideration to the 
ors previously enumerated. We then will apply the equa- 

to every task in the occupational area. By this method, 
ay be able to identify tasks not being trained which would 
rained if the judgment process had teen systematic. In 
same manner, we may be able to identify tasks being 
ned which the policy equation suggests should not be 
ned. The rationale for this approach will be discussed in 

detail when we view the final reports generated in Step 7. 

It was hypothesized that certain of the task rating 
ors might be related to the criterion in a non-linear 
ion. In order to test this hypothesis, we needed to 
re certain vectors for introduction into the regression 
tions. This was accomplished in Step 5 by the COMGEN 
ram. 

At this point, we have available on the JOB DESCRIPTION 
all variables required for computing the final regres- 
equation in Step 6. This is accomplished using the 
DR program. In the same run, the computed regression 
tits are applied to predictors using the PREFAC program, 
this vector is written onto the JOB DESCRIPTION file using 
TSKFAC program. 

The final results are displayed by reports generated in 
7 using the FACSUM program. I'm sorry that vugraphs are 
restricted in space to permit a full display of FACSUM 
, but I will present information extracted from one run 
ables 6 and 7. First let's look at Table 6. 

In the first column, the FACSUM program has listed the 
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titles for the eighteen tasks selected for the example using 
the operator variable generated in Step 4 by the TSI&AT pro- 
gram. Column 2 displays the training priority indexes gener- 
ated by the TSKCOR and PREFAC programs in Step 6. Notice 
that all information in Table 6 has been ordered from high to 
low on the training priority index. The FACSUM program pro- 
vides the investigator with the option of ordering information 
from low to high or high to low on any vector — or to list data 
in duty/task order. 

Column 3 identifies tasks currently being taught in the 
academy by a value of "1," and tasks which are not being 
taught by a value of "0." This vector was introduced onto the 
JOB DESCRIPTION file in Step 2 using th* TSKFAC program. 

Column 4 indicates the percent of patrolmen who have been 
on the job 12 months or less performing each task. This was 
generated in Step 3 using the JOBSPC program. 

Columns 5, 6, and 7 present the mean task-factor ratings 
on Delay Tolerance, Consequences of Inadequate Performance, 
and Learning Difficulty. These mean values were computed in 
Step 1 using the REXALL program, and were introduced onto the 
JOB DESCRIPTION file using the TSKFAC program. 

Column 8 reports the differences between the values in 
Columns 6 and 7, and h^'* been included in the table to illus- 
trate one of the many : bilities of the FACSUM program. 
High positive values in ^iis vector identify tasks which are 
easy to learn, but for which the consequences of inadequate 
performance are severe. Examples are "advising suspects of 
their legal rights 11 and "conducting frisk searches. 11 High 
negative values on the vector identify difficult tasks with 
minor consequences for inadequate performance, examples being 
"presenting speeches" and "conducting airborne traffic 
surveillance. " 

Column 9 presents the average grade values computed by 
the AVGPCT program in Step 3. it reveals that "Inspect Men 
at Roll Call" is a sergeant or lieutenant task, and not 
appropriate for inclusion in the academy entry-level course. 

Now let's go back and look at Column 3, which indicates 
whether each task is or is not included in the present course. 
Remember that the tasks are ordered in terms of values or the 
Training Priority Index. Notice that training is already 
being given on seven of the top eight tasks. The one excep- 
tion is the third task listed: "Engage in High-Speed Pursuit 
Driving." No training is presently included in the course on 
• 
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this fc&sk. The equation suggests that training on this task 
should he introduced if curriculum designers are to be sys- 
tematic :.n their decision process. The fact /r data reveal 
that C9% of graduates encounter this task during the first 
year after graduation. It has the lowest Task Delay Toler- 
ance among all of the tasks on this chart. This means that 
when the task is encountered, performance must begin immedi- 
ately, or the opportunity to perform it rapidly perishes. 
There is no time to read a manual or consult a friend. The 
consequences of inadequate performance are severe. I'm not 
sure of the exact values, but I recall reading in a recent 
newspaper article that about 50% of high-speed chases end with 
an accident, and that a significant number end with a fatality. 
Data on the chart show the task to be difficult to learn, and 
indicate that it is appropriate for consideration for 
inclusion in the academy course. 

Now let me qualify this statement. The equation actually 
reflects the need for entry-level training on each task. It 
does not address the question as to where that training should 
be given. It is possible that this particular task could be 
taught more economically in the job setting. Be that as it 
may, the equation does reflect a need for training. 

Looking at the bottom of the chart, we find that train- 
ing is currently being given on two of the bottom seven tasks. 
The bottom task, "Prepare and Present Speeches," is performed 
by only seven percent of academy graduates during their first 
year on the job. While the task is rated as being difficult 
to learn, performers are seldom under time pressure (Task 
Delay Tolerance = 7.8), and supervisors rate the consequences 
of inadequate performance as being minor. For somewhat the 
same reasons, the equation raises questions concerning the 
training being given on "Conduct Airborne Traffic Surveillance," 
which is performed by only 10% of the graduates. 

Table 7 presents an extract of another FACSUM output 
obtained in the same CODAP run. In this case, a difference 
score was generated between the observed and predicted 
training priorities, and the tasks are ordered on that vector. 
In accordance with user specifications, those tasks with near- 
equal observed and predicted priorities were suppressed from 
the printout, leaving only those tasks for which training 
decisions are required. 

I'm sure that you appreciate the impossibility of 
adequately describing the flexibility and utility of the new 
task-oriented CODAP programs in the few minutes available to 
me at this conference. 



281 



17 



0) 

CO 

O 

0) 
00 
M 

O 

a 



CO 
CO 
O 
P-i 

J-i 
O 
UH 

00 

o 



CO 

c 
a) 



o 
o 
a) 



a) 

iH 

5 



saouanbasuoQ 



CO 
CO 
H 

















On 


















CO 


Cm 










Cn 


CO 




rH 








CN 


m 


ON 


• 


* 








* 




* 














m 


*n 




Cn 


*»c 


¥¥¥¥¥¥¥¥ * 


in 






* 


* 


*fc 




•SC 


* 


* 








•Jc 


O 


* 






CO 






•♦6 


O 


* 












*K 


m 


* 












*lc 


• 


* 








rH 


rH 


*lc 




* 




CN 


00 


* 


* 


•♦6 




* 


* 




* 


rH 




*|C 


CO 


* 


in 


cn 








•♦6 


rC 


* 












*#c 


H 


* 












*K 




* 








on 


CN 


*K 


00 






/—^ 

w 






CO 


•Jc 


00 


* 


rH 


rH 








•K 


<u 


* 












•K 




* 












•K 




* 








O 


CN 


■X 


rC'CO 


* 


/»— <\ 
w 


CN 




ON 


ON 


*K 


CJ "H 


* 


/»— <» 
w 


rH 


CN 


o 




*iC 


•H 


* 


%.r\ 


ON 


O 


• 


• 


•K 




* 


* 






rH 




•K 






1 


r 


rH 






•K 




* 






1 






•K 


H rH 


* 












•K 


o q 


* 










o 






* 


rH 


rH 


rH 










* 












* 




* 














0) H 


* 








O 


CN 


* 


4J 


* 


o 


00 


CO 


ON 


on 


* 


4J 1 


* 


o 


00 


CN 


o 


m 


* 


•H 


* 




o 


O 






* 




* 








rH 




* 


53" 


* 






f 






* 




* 












* 


00 v£> 


* 












* 


o 


* 












* 


CO 1X4 


* 




0) 




60 






co 2 


* 




CJ 




c 






H CO 


* 




c 




•H 






H 


* 




CO 






00 






* 




rH 






a) 






* 




rH 




rJ 


CO 


* 




* 




•H 




Q 


CO 




S3 






cu 






o 




CJ 


* 








4J 






c 


* 






00 


•H 


rH 




0) 


* 


00 




cu 


0 


CO 


* 


u 


* 


a) 


CO X 


CO 


c 


* 


0) 


* 


00 




CJ 


rJ 


•H 


* 


MH 


* 


CO 


a 


0) 






* 


MH 




0) 


*H 


0) 






* 


•H 






Cu 






* 


Q 


* 


4J 




CO 




O 


* 




* 


•H 


CO 





a) 
a) c 

CU iH 

co 

i co 

rC 4J 

00 0 

•h a) 
* § 

G 4J 
•H CO 
4J 

a) co 
oo 

co a) 

00 r* 

C CO 
W H 



* ¥¥¥¥¥¥¥¥ * 



18 



o 

CO 



00 

*| 

o c 

rC O 

CO CJ 



4J 
0 

G) 
CO 

a) 

r4 

C 

CO 

0) 
M 
CO 
Cu 

a) 

M 



282 



ERLC 



